feat: add metrics storage-read-bytes for parquet and fix clean inte test#439
Conversation
…fix clean inte test
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds byte-level Parquet read metrics (raw-input-bytes / storage-read-bytes) by plumbing a shared counter from the Arrow input stream adapter into the Parquet batch reader, and adjusts commit/IT behavior around uncertain commit outcomes.
Changes:
- Track and surface Parquet read byte metrics via
ArrowInputStreamAdapter→ParquetFileBatchReader→ParquetMetrics. - Update Parquet-related unit tests to pass the new parameter and assert the new metrics.
- Change commit retry behavior on uncertain commit failures and update affected integration/unit tests.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/inte/clean_inte_test.cpp | Re-enables an integration test previously disabled for IO exception scenarios. |
| src/paimon/format/parquet/predicate_pushdown_test.cpp | Updates Parquet reader creation to pass the new raw-bytes counter parameter. |
| src/paimon/format/parquet/parquet_reader_builder.h | Extracts raw input byte counter from stream adapter and passes it into the batch reader. |
| src/paimon/format/parquet/parquet_format_defs.h | Defines new Parquet metrics keys for byte-level read tracking. |
| src/paimon/format/parquet/parquet_file_batch_reader_test.cpp | Wires raw-bytes counter through tests and adds assertions for the new metrics. |
| src/paimon/format/parquet/parquet_file_batch_reader.h | Extends API to accept a raw-bytes counter and sets the new metrics on GetReaderMetrics(). |
| src/paimon/format/parquet/parquet_file_batch_reader.cpp | Implements new constructor/creator signature to store the raw-bytes counter. |
| src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp | Updates tests to pass the new raw-bytes counter parameter. |
| src/paimon/core/operation/file_store_commit_impl_test.cpp | Adjusts expectations for commit failure/uncertain commit and adds FilterAndCommit verification. |
| src/paimon/core/operation/file_store_commit_impl.h | Removes CheckCommitted and simplifies TryCommitOnce signature. |
| src/paimon/core/operation/file_store_commit_impl.cpp | Stops internal retry-on-exception path and returns an error instructing caller to use FilterAndCommit. |
| src/paimon/common/utils/arrow/arrow_input_stream_adapter.h | Adds shared atomic counter accessor for raw input bytes. |
| src/paimon/common/utils/arrow/arrow_input_stream_adapter.cpp | Increments raw input bytes counter on reads (sync and async). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
zjw1111
left a comment
There was a problem hiding this comment.
Requesting changes for three remaining correctness and observability issues. The previously reported missing ParquetFileBatchReader::Create argument has been fixed.
Purpose
Linked issue: close #xxx
This change bundles two independent items:
1. Parquet byte-level read metrics (feature)
Add two counters so callers can observe how many bytes a Parquet read actually consumes:
parquet.read.storage-read-bytes— physical bytes read from storage.2. Revert #397 commit internal-retry / dedup (fix)
#397changed the commit-exception branch inTryCommitOncefromreturn Status::Invalid(...)toreturn false, driving an internal retry loop plus aCheckCommitted/retry_start_snapshot_iddedup. This introduced flakiness in
CleanInteTest.TestDropPartitionAndExpireSnapshotWithIOExceptionand diverged from Java, which has no equivalent
RetryCommitResulttype system in this C++ codebase.This reverts to the pre-#397 behavior:
retry_start_snapshot_idfrom theTryCommitloop and theTryCommitOncesignature.CheckCommittedmethod (definition and declaration).return Status::Invalid("You need call FilterAndCommit to retry commit for exception. ", ...).Rationale: on a commit exception the outcome is uncertain (an atomic write may have timed out but
actually landed). Retrying internally can produce a duplicate snapshot. Safe retry belongs to the
upper layer via
FilterAndCommit, which dedups byCommitIdentifier(); a bareCommitretry has nosuch dedup and would create a duplicate. Returning
Status::Invalidmakes that contract explicit andkeeps
guard.Release()so freshly written meta files are not cleaned up.Tests
TestCommitWithAtomicWriteSnapshotTimeoutAndActuallySucceedrestored to its pre-feat(commit): align functionality with java paimon commit #397 assertions:the initial
Commitreturns an error (ASSERT_NOK) while the snapshot has physically landed, andrecovery goes through
FilterAndCommit, which idempotently filters the already-committedidentifier (
num_committed == 0).CleanInteTest.TestDropPartitionAndExpireSnapshotWithIOException— re-enabled (removed theDISABLED_prefix); the CI-friendly randomized loop step is kept. Verified deterministically byrunning the loop exhaustively (
i += 1, 0..1000) locally: PASSED (~14 min), confirming the revertroot-causes and eliminates the flakiness rather than lowering its probability.
parquet_file_batch_reader_test.cpp,page_filtered_row_group_reader_test.cpp,predicate_pushdown_test.cpp(Create call sites now passthe
storage_read_bytescounter).API and Format
src/paimon/...); no headers underinclude/paimon/are modified.parquet.read.storage-read-bytes.ParquetFileBatchReader::Create/constructor gain a requiredstorage_read_bytesparameter;FileStoreCommitImpl::TryCommitOncedropsretry_start_snapshot_idand
CheckCommittedis removed. All call sites are updated in this change.Documentation
No new user-facing feature documentation. The change adds observability counters and reverts an
internal retry behavior; no doc pages are introduced.
Generative AI tooling